Developing REST Services and Clients

Before the web and distributed systems that now permeate the cloud space, standards for communicating between systems were not in widespread use. This communication is often called an RPC. This simply means that a program on one machine has a method to call a function running on a different machine and receive any output.

Monolithic applications were the norm and servers tended to either be silo'd per application and vertically scaled or were run as jobs on larger, more specialized hardware from companies such as IBM, Sun, SGI, or Cray. When systems did need to communicate with each other, they tended to use their own custom wire formats, such as what you would see with Microsoft SQL Server.

With the web defining the internet of the 2000s, large monolithic systems could not provide the compute power behind services such as Google Search or Facebook at any reasonable cost point. To power these services, companies needed to treat large collections of standard PCs as a single system. Where a single system could communicate between processes using Unix sockets or shared memory calls, companies needed common and secure ways to communicate between processes running on different machines.

As HTTP became the de facto standard for communication between systems, RPC mechanisms of today use some form of HTTP for data transport. This allows the RPC to transit systems more easily, such as load balancers, and easily utilize security standards, such as Transport Layer Security (TLS). It also means that as the HTTP transport is upgraded, these RPC frameworks can leverage the hard work of hundreds if not
thousands of engineers.

In this lesson, we are going to talk about one of the most popular RPC mechanisms, REST. REST uses HTTP calls and whatever messaging format you want, although the majority of cases use JSON for messaging.

REST for RPCs#

Writing REST clients in Go is fairly simple. Chances are that if you have been developing applications in the last 10 years, you have either used a REST client or written one. Cloud APIs for services such as Google Cloud Platform's Cloud Spanner, Microsoft's Azure Data Explorer, or Amazon DynamoDB use REST to communicate with the services via their client libraries.

REST clients can do the following:

• Use GET, POST, PATCH, or any other type of HTTP method

• Support any serialization format (although this is normally JSON)

• Allow for data streaming

• Support query variables

• Support multiple versions of an API using URL standards

REST in Go also has the luxury of not requiring any framework to implement on the server side. Everything that is required lives in the standard library.

Writing a REST client #

Let's write a simple REST client that accesses a server and receives a Quote of the Day (QOTD). To do this, the server has the following endpoint using POST – /v1/qotd. First, let's define the message we need to send to the server:

Let's talk about what each of these does:

• getReq details the arguments to the server's /v1/qotd function call.

• getResp is what we expect as a return from the server's function call.

We are using field tags to allow conversion from lowercase keys into our public variables that are capitalized. For the encoding/json package to see these values for serialization, they must be public. Private fields will not be serializable:

This defines a custom error type. This way, we can store error codes to return to the user. This code is defined next to our response object, but it isn't used until much later in the code we are defining.

Let's now define a QOTD client and a constructor that does some basic checks on the address and creates an HTTP client to allow us to send data to the server:

The next step is to make a generic function for making REST calls. Because REST is so open-ended, it is hard to make one that can handle any type of REST calls. A best practice to use when writing REST servers is to only support the POST method; never use query variables and simple URLs. However, in practice, you will deal with a wide variety of REST call types if you don't control the service:

The code above works as follows:

• Lines 3–7: Check our context for a deadline:

  • If it has one, it is honored.

  • If not, a default one is set.

  • cancel() is called after the call is done

• Line 10: Marshals a request into JSON

• Lines 17–22: Creates a new *http.Request that does the following:

  • Uses the POST method.

  • Talks to an endpoint.

  • Has io.Reader storing the JSON request.

• Lines 28–31: Uses the client to send a request and get a response

• Line 34–37: Retrieves the response from the body of http.Response

• Line 40: Unmarshals JSON into the response object

We will notice that req and resp are both interface{}. This allows us to use this routine with any struct that will represent a JSON request or response. Now, we'll use that in a method that gets a QOTD by an author:

The code works as follows:

• Line 3: Defines an endpoint for our get function on the server

• Lines 10–16: Calls our restCall() method, which does the following:

  • Uses path.Join() to unite our server address and URL endpoint.

  • Creates a getReq object as the req argument of restCall().

  • Reads the response into our resp response object.

  • If *http.Client returns an error, we return that error.

  • If resp.Error is set, we return it.

• Line 18: Returns the response's quote

To see this running, refer to the code below:

We have shown how to make a base REST client here using HTTP POST calls and JSON. However, we have only scratched the surface of making a REST client. You may need to add authentication to the header in the form of a JSON Web Token (JWT). This used HTTP and not HTTPS, so there was no transport security. We did not try to use compressions such as Deflate or Gzip.

While using http.Client is easy to do, we may want a more intelligent wrapper that handles many of these features for us. One that is worth looking at would be resty, which can be found here on GitHub.

Writing a REST service#

Now that we have a client written, let's write a REST service endpoint that can receive the request and send the user the output:

This code does the following:

• Creates the server struct, which will act as our server

• Uses *http.Server to server HTTP content

• Has quotes, which stores authors as keys and values that are a slice of quotes

Now, we need a constructor:

This code does the following:

• Line 1: Creates a newServer constructor:

  • This has an argument of port, which is the port to run the server on.

• Lines 2–8: Creates a server instance:

  • Makes an instance of *http.Server running at :[port].

  • Populates our quotes map.

• Lines 10–11: Adds *http.ServeMux to map URLs to methods.

Note: We'll create the qotdGet method in a moment.

• Lines 18–20: Creates a method called start() that will start our HTTP server.

*http.ServeMux implements the http.Handler interface that is used by *http.Server. ServeMux uses pattern matching to determine which method is called for which URL.

Now, let's create the method to answer our REST endpoint:

This code does the following:

• Implements the http.Handler interface

• Lines 14–18: Reads the HTTP request body and marshals it to our getReq:

  • This uses HTTP error codes with http.Error() if the request was bad.

• Lines 23–30: If the request did not contain an "author," randomly chooses an author's quotes

• Lines 31–52: Otherwise, find the author and retrieve their quotes:

  • If that author did not exist, respond with getResp containing an error.

• Lines 59–65: Randomly chooses a quote and returns it to the client

We can see the entire executable code below:

Now, we have a REST endpoint that can answer our client's RPCs. This just scratches the surface of building a REST service. We can build authentication and compression on top of this, performance tracing, and so on.

To help with bootstrapping features and removing some boilerplate, a few third-party packages, such as Gin and Revel might be helpful.